Transactional computer systems are becoming larger and more complex. Telephone companies have implemented transaction processing to improve efficiencies in their operational support systems. Typically, users interact with these transaction processing systems by entering data in fields on a CRT screen and then expecting a response in other fields on the screen. As new software is developed, these software systems have to be tested before they are delivered to end users.
Software testers test by developing test cases that specify combinations of screen inputs which are entered into the system to see if the response of the system is appropriate as a result of the inputs. One method of testing would be to create test cases for all combinations of all input values on the screen. For most systems, because of the large number of screens with multiple fields each with multiple values, the number of test cases that can be generated is extremely large. Accordingly, the cost of conducting such tests could be exorbitant. Therefore, in the current art the number of test cases is reduced by taking sample values for each field; such as the minimum value, a maximum value, and some intermediate value for a numeric field. Even so, the cost of conducting screen testing in the current art can increase the cost of software development by up to 30%. These problems are not unique to screen based software systems. In the area of protocol conformance testing, protocols and features have typically been modeled as finite state machines. However, the number of test cases to test a finite state machine with n states is a polynomial in n. Since protocols typically have thousands of states, and may have millions, traditional finite state machine based testing produces an unreasonable number of tests. The example that follows highlights some of the problems.
Assume a system with one screen and n fields on this screen. Suppose also that after an analysis of the system requirements used to create the software, only a limited number of inputs (i) is considered for testing. Then the number of possible test cases, each containing values for all fields on the screen is i.sup.n. This can be quite large even for a single screen. As an example, suppose only three values (i=3) are to be considered for each field and there are 13 fields (n=13) on the screen then the number of exhaustive test cases are 3.sup.13 =1,594,323. Obviously, this number is very large and exhaustive testing is far beyond anyone's available resources. Thus, typical testers find various ways of cutting down the number of combinations, e.g. by gut feel, or by just trying out all the values one field at a time, holding all other fields at default values. Unfortunately, this method of default testing will not take into account interactions, or dependencies between various fields.
Some researchers in the field have examined the application of experimental design techniques used for planning experiments to the area of software testing (Phadke, M. S., "Quality Engineering Using Robust Design", Prentice Hall, Englewood Cliffs, N.J., 1989). Specifically, Pahdke proposes the use of orthogonal array designs for testing software. Orthogonal array designs are test sets such that for any pair of fields all combinations of input levels occur exactly an equal number of times. As an example, consider the situation where one screen has three fields (n=3) with two possible inputs per field (i=2). An exhaustive test of all possible combinations would result in 8 test sets as shown in Table 1 below.
TABLE 1 ______________________________________ Exhaustive Test Array FIELDS TEST NO. F.sub.1 F.sub.2 F.sub.3 ______________________________________ 1 1 1 1 2 1 1 2 3 1 2 1 4 2 1 1 5 1 2 2 6 2 1 2 7 2 2 1 8 2 2 2 ______________________________________
Using experimental design techniques, a corresponding orthogonal array of the same fields and values results in 4 test cases as shown in Table 2 below.
TABLE 2 ______________________________________ Orthogonal Test Array FIELDS TEST NO. F.sub.1 F.sub.2 F.sub.3 ______________________________________ 1 1 1 1 2 1 2 2 3 2 1 2 4 2 2 1 ______________________________________
Table 2 depicts every pair of inputs exactly once and has only four test cases. This is a 50% reduction in test cases from the fully enumerated set of test cases shown in Table 1.
While orthogonal arrays provide test sets that cover every pair of inputs with many fewer test cases than the exhaustive test set, this prior art solution has a number of problems. First and foremost, not every combination of fields and inputs has a corresponding orthogonal array. For example, when there are 7 fields and each field has 6 possible inputs, an orthogonal array does not exist. Another problem with the orthogonal array is that they are difficult to construct; there is no single unified way of generating an orthogonal array. This difficulty is exacerbated when there are an unequal number of values for each of the fields. Orthogonal arrays are also wasteful in that they require each pair of inputs to occur exactly the same number of times. This property of balance is often necessary in statistical experiments where equal precision is required. However, in software testing it is unnecessary to require an equal number of replicates.
Therefore, an object of our invention is an automated method and system that overcome the limitations in the prior art and incorporate interactions between fields while reducing the number of test cases necessary to robustly test software systems.